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Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) OR THIRTY (30) DAYS. 
WHICHEVER IS LONGER. FROM THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1.136(a). In no event, however, may a reply be tirhely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 1 33). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1 )KI Responsive to communication(s) filed on 09 October 2007 . 
2a)K This action is FINAL. 25)0 This action is non-final, 

3) 0 Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 11. 453 O.G. 213. 

Disposition of Claims 

4) ^ Claim(s) 37 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) n Claim{s) is/are allowed. 

6) K Claim(s) 37 is/are rejected. 

?)□ Claim(s) is/are objected to. 

8) n Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) n The specification is objected to by the Examiner. 

10) D The drawing(s) filed on is/are: a)n accepted or b)n objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 
Replacement drawing sheet(s) including the correction is required If the drawing(s) Is objected to. See 37 CFR 1.121(d). 

1 1) 0 The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 119 

12) n Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (f). 
a)n All b)n Some * 0)0 None of: 

1 .□ Certified copies of the priority documents have been received. 

2. n Certified copies of the priority documents have been received in Application No. . 

3. n Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 
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DETAILED ACTION 
Remarks 

1 . In response to Applicant's Amendment filed on October 9, 2007, claim 37 is pending in 
the application. 

Claim Objections 

2. Claim 37 is objected to because of the following informalities: 

Claim 37 recites the limitation "the image contents" in line 13. There is insufficient 
antecedent basis for this limitation in the claim. 

Claim Rejections - 35 JJSC §103 

3. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 

obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

4. Claim 37 is rejected under 35 U.S.C. 103(a) as being unpatentable over Sieurionsson et 
aL (U.S. Pub. No. 2002/0107829 Al) in view of Kobus Barnard and David Forsyth . Learning the 
Semantics of Words and Pictures. IEEE 2001 (From hereon in Barnard et al.\ and further in 
view of Liu etal. (U.S. Patent No. 6,970,860 Bl). 



As to claim 37, Sieurionsson et al. discloses a method comprising: 
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obtaining one stored images wherein the one or more stored images are stored within an 
image database, wherein the stored images are associated with at least one of a multiplicity of 
keywords (See page 3, paragraph 0030); 

obtaining an image and comparing the image to the stored images to identify one or more 
similar images wherein image similarity is based on at least two factors, wherein one of the at 
least two factors is closeness in time of image acquisition (See Figure 9A); 

producing at least two primary keywords based on the keywords associated with the 
similar images (See page 4, paragraphs 0042-0047); 

determining at least two primary associations with the primary keywords (See page 3, 
paragraph 0037); 

producing at least two secondary associations that identify at least two finer scale 
keywords wherein the secondary associations are based on the keyword and the primary 
associations wherein the database indicate that the finer scale keywords are keywords that are 
associated and occur when at least one of the primary keywords occurs (See page 3, paragraph 
0037, and see page 4, paragraphs 0045-0046); 

presenting the primary keywords, the finer scale keywords, and a new keyword selection 
to a user (See page 4, paragraph 0048); 

obtaining one or more accepted keywords from the user wherein the accepted keywords 
comprise any one of or combination of primary keywords, the finer scale keywords, and one or 
more new keywords entered via the new keyword selection (See page 4, paragraphs 0049-0050); 
and 
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storing the image within the image database and in association with the accepted 
keywords (See page 3, paragraph 0038); and 

updating the keyword wherein the keywords further comprise the accepted keywords 
(See page 4, paragraph 0048). 

Sigurionsson et al. does not explicitly teach the association of tags to be statistical as 
claimed to be wherein a multitude of keyword statistics are stored in a database statistics module, 
and wherein the keyword statistics indicate a probability that one of the keywords is associated 
with the one of the stored images whenever another one of the keywords is associated with the 
same stored image; likelihoods of keywords; and updating the keyword statistics. Sigurionsson et 
al teaches multiple tags (keywords) associated with an image and with each other (See page 4, 
paragraph 0047); and updating the keyword database (See page 4, paragraph 0048). 

Bamard et al. teaches wherein a multitude of keyword statistics are stored in a database 
statistics module, and wherein the keyword statistics indicate a probability that one of the 
keywords is associated with the one of the stored images whenever another one of the keywords 
is associated with the same stored image (See page 412, column 2); and 

likelihoods of keywords (See page 413); 

updating the keyword statistics (See page 414, column 2). 

It would have been obvious to one having ordinary skill in the art at the time the 
invention was made to modify the teachings of Sigurionsson et al. with the teachings of Bamard 
et al. to include association of multiple keywords in the same category (i.e. primary scale, finer 
scale) using statistics and likelihood of occurrence as it allows for faster automated classification 
of images for better search as taught in Bamard et al. page 408, column 1 . 
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The combination of Sigurionsson et al, with Barnard et aL although teaches associating 
keyword with images, does not exphcitly teach wherein another one of the at least two factors is 
based on the image contents as determined by a content based image retrieval module, and 
wherein no keywords are currently associated with the image. 

Liu et al. teaches wherein another one of the at least two factors is based on the image 
contents as determined by a content based image retrieval module, and wherein no keywords are 
currently associated with the image (See Liu et al. column 6, lines 11-18, and see Liu et al. 
column 6, lines 30-36, and Liu et al. column 6, lines 61-65). 

It would have been obvious to one having ordinary skill in the art at the time the 
invention was made to modify the teachings of Sigurjonsson et al. as defied with the teachings of 
Liu et al. to include wherein another one of the at least two factors is based on the image contents 
as determined by a content based image retrieval module, and wherein no keywords are currently 
associated with the image because for more accurate annotations of images (See Liu et al. 
column 1, lines 37-45. 

Response to Arguments 
5. Applicants arguments filed on October 9, 2007 have been fully considered but they are 
not persuasive. 



Applicant's arguments on page 6, and bottom of page 9 that " Sigurionsson et al. teaches 
hierarchical tree which can only be traversed bottom to top while the opposite can't hold thus 
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does not teach or suggest the claimed invention of bi-directional associated probability" is not 
found to be persuasive. 

It appears that in contrast to Applicant's remarks, Applicant's specification what is 
defined as "primary keywords" and "finer scale" keywords are in fact "root category heading" 
and "sub-category entities" (see Published version paragraph 0033) thus would not follow 
applicant's rational and argimient that frequency of occurrence and probability of association 
would need to be bi-directional. It's in fact opposing to the specification which explicitly state 
that the probability is history of no. of occurrence of each keyword within a specified category 
(see Published version paragraphs 0023-0024). The claim language itself also is in conflict of 
paragraph 0025 where it is explicitly stated that "Primary image category" and "finer scale 
keywords" NOT both being keywords and NOT both having multiple probabilities. In fact, no 
where in the specification does it specify the number of associations nor the number of keywords 
attached to any image. What appears to be calculated is a primary category and the frequency of 
occurrence of keywords within that category thus making it a high probability of being 
associated. Also in contrast to Applicant's arguments, CBIR is the only initial contributor to the 
image similarity when a new image is first loaded in the system as taught in specification 
published version paragraph 0029, database statistic and learning algorithm will then take effect 
after those similar images were shown to the user; thus significantly different fi-om what is being 
claimed and argued. In fact, the applicant's specification parallel the prior art combination of 
Sigurionsson et al. and Barnard et al. cited by the Examiner in that hieratical linked keywords are 
formed and statistics are calculated separated by CBIR and layered on top of the hierarchal 
structure accordingly to query retrieval feedback statistic. 
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Claim 37 language of "producing at least two primary keywords based on the keywords 
associated with the similar images; determining at least two primary associations with the 
primary keywords; producing at least two secondary associations that identify at least two finer 
scale keywords wherein the secondary associations are based on the keyword and the primary 
associations" has no requirenmt, support, nor description in the specification. 

What is essential to Applicant's invention and missing from the claims is the concept of 
high-hit ratio whereby the calculated statistics are based on as taught in paragraph 0017. 

In response to Applicant's argument that "there examiner has not offered a motivation to 
combine and did not follow KSR rational to provide evidence to suggest the combination of 
references teach the functionality claimed" is not deemed persuasive. 

The Examiner has properly provided a motivation from teachings in the secondary 
reference. Barnard et al. background teaches various problems dealing with current image 
classification and retrieval systems including ones similar to Sigurionsson et al. *s. while the 
Examiner opted to cite page 408, colunm Ito highlight Barnard et al. 's motivation to improve 
prior art systems by "integrating statistical semantics learned based on queries" instead of 
waiting for associative assignment of tags thus an improvement to Sigurjonsson et al. 's tagging 
by association only system. 

The Supreme court KSR decision (reasoning) is meant to be given as examples of 
possible ways of combining references and NOT as a test for references to meet each provision 
in order to be combinable. In this case, the Examiner provided motivation in the secondary 
reference and emphasizes following the Supreme court's reasoning, to one of ordinary skill in the 
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art, the combination of Sigurionsson et al. and Barnard et al. cited is certainly obvious to try to 
combine the semantic associations relative to images of Sigurionsson et al. with content based 
image retrieval techniques holding statistical image keyword calculations of Barnard et al. . 
Hierarchical keywords assignment and linkage as taught by Sipurjonsson et al. is clearly found in 
Applicant's specification, and statistically calculation for image similarity used in content 
retrieval systems as taught by Barnard et al. is too found in Applicant's specification. 

Note: although a new rejection is presented to advance prosecution. Applicant's argument 
that CBIR is well knovm in the art contradicts the novelty described by the application's own 
disclosure paragraph 0024 since it is clearly stated that Applicant's CBIR is not your 
conventional system. Furthermore, Barnard et al. clearly teaches content based image retrieval. 
Thus the combined references do in fact teach CBIR 

6. Although the Examiner completely disagrees with applicant's argument and specifically 
raises the issue that they have no merits in light of the specification, the Examiner has opted to 
introduce the Liu et al. (U.S. Patent No. 6,970,860 Bl) reference to show exact teachings of the 
claimed invention and thus expedite prosecution. 

Conclusion 

7. THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1.136(a). 
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A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed xmtil after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1 .136(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the mailing 
date of this final action. 

8. The prior art made of record and not relied upon is considered pertinent to applicant's 
disclosure. 

Caid et al. (U.S. Patent No. 6,760,714 Bl) teaches co-occurrence of each meta descriptor 
to another in a class of images. 

Li et al. (U.S. Patent No. 6,175,829 Bl) teaches providing user with alternative terms to 
be used in annotating images based on co-occurrence values in a database. 

Liu et al. (U.S. Patent No. 6,970,860 Bl) teaches CBIR and text associations. 

De Bonet (U.S. Patent No. 5,819,288) teaches statistically based image group descriptor. 

Jain et al. (U.S. Patent No. 5,9 11, 139) teaches image vector database calculations. 

Vaithilineam et al. (U.S. Patent No. 6,411,724 Bl) teaches using meta descriptors in 
query by example image retrieval. 

David et al. (U.S. Patent No. 7,010,144 Bl) teaches associating data with images in a 
database. 
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Thomas et al. (U.S. Patent No. 7,043,094 B2) teaches semantic feature accessibility in a. 
database. 

For complete list of relevant art, see PTO form 892. 

9. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Neveen Abel-Jalil whose telephone number is 571-272-4074. 
The examiner can normally be reached on 8:30AM-5 :30PM EST. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor. Christian Chace can be reached on 571-272-4190. The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EEC) at 866-217-9197 (toll-free). If you would 
like assistance from a USPTO Customer Service Representative or access to the automated 
information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 




Primary Examiner 
January 1, 2007 



